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Notes in Brief 

Institutions of higher education are faced with challenges when implementing an 
assessment initiative. These challenges include constraints on time and resources, 
the demands of specialized accreditation, and faculty motivation to comply. In 
addition, the communication of the expectations around assessment must be clear 
to all constituents in order for such initiatives to be successful. Communication 
begins with defining what the institution envisions to be the core elements 
that must be seen in a plan regardless of disciplinary uniqueness or specialized 
accreditation. When these core elements are developed and evaluated in a 
systematic and generalizable manner, institutions can move beyond the collection 
of quantitative data regarding the number of programs that are in compliance with 
the assessment mandate—i.e., merely counting the number of programs that report 
outcomes, measures, results, and action plans. Institutions can begin to conduct 
qualitative reviews of program-based assessment plans, identifying the clarity and 
value of plan components with regard to their usefulness for the improvement of 
student learning. Using the work of one regional comprehensive public university, 
this case study will demonstrate guiding principles for institutional success in both 
developing a generalizable assessment initiative and communicating this important 

work to institutional constituents. 


Guiding Principles to Impact an Institution- 
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In light of increasing expectations of accountability for student learning 
outcomes, student learning assessment continues to be of central importance to 
institutions of higher education (IHEs). Institutions are looking for ways to implement 
successful approaches for assessment or the assurance of student learning to ensure it is 
taken seriously by faculty, and is integrated into the fabric or culture of the institution 
(Maki, 2004). Specifically, institutions want to actively engage faculty in the work of 
student learning assessment in order to help them see the value of this work for local, 
curricular, and pedagogical purposes, beyond merely meeting the requirements of 
external constituents. A recent Chronicle of Higher Education article entitled “Giving 
Assessment a Fighting Chance” (Havens, 2013) discusses several guiding principles 
for institutions to consider when implementing institution-wide change. These guiding 
principles include the following: (a) avoid conducting institution-wide change under the 
“gun” of an accreditation deadline; (b) avoid introducing a brand new system or process 
immediately and, instead, take stock of what programs are already doing and processes 
already in place; (c) ensure some level of quality control so that plans are not created 
or implemented in an inefficient or ineffective way; and (d) provide appropriate support 
to enable the work to be accomplished (particularly time and money). The purpose of 
this case study is to outline an institution-wide initiative using these guiding principles 
and to present data that demonstrate a positive change in the quality of student learning 
assessment as a result of that initiative. 

Avoid Conducting Institution-Wide Change Under 
the Gun of an Accreditation Deadline 


While accreditation visits are the impetus for much of the student learning 
assessment activity that takes place on campuses, it must not be the only motivation for 
activities to occur (Lubinescu, Ratcliff, & Gaffney, 2002). The time spent preparing for 
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accreditation visits allows institutions to reflect on their current practices and validate their 
work through evidence. During a recent regional accreditation visit, the institution discussed 
in this report was successful in satisfying the standard related to student learning assessment. 
Upon further reflection, however, it became clear there was unevenness in the overall state 
of student learning assessment on campus. In essence, the institution became its own biggest 
critic. There clearly were pockets of good, even excellent, assessment practices, but there was 
significant variability across programs and colleges with regard to the quality of assessment 
practices. Building on the momentum of the regional accreditation visit, the institution 
implemented a campus-wide assurance of student learning (ASL) initiative to encourage 
programs to critically evaluate, and revise if necessary, their program-level student learning 
assessment plan(s). 


Avoid Introducing a Brand New System 

It was important to listen to a variety of constituents on campus to fully understand 
existing processes and to avoid duplication of previous or current efforts. At this institution, 
assessment of student learning is overseen by an appointed faculty member (provided 
half-time release from the office of the provost), who has the title of Faculty Associate for 
Teaching, Learning and Assessment, and an associate dean within each of five academic 
colleges (one of the academic colleges is broken into two schools for a total of six associate 
deans). Individual departments within each college have faculty-appointed assessment 
coordinators who provide oversight of the process for individual programs and serve as 
liaisons between department faculty, department chairs, and associate deans. At the time 
we were first planning the ASL initiative, departments, faculty, and administrators had 
just worked tirelessly to put together reports related to student learning assessment for the 
regional accreditation site visit. Thus, it was very important to listen to their views about the 
status of assessment efforts in an attempt to best address their needs. Thus, we first initiated 
a conversation with the associate deans and assessment coordinators to try to understand 
their perspective about the processes in place and to hear their collective thoughts of the 
limitations and/or stumbling blocks associated with those processes. 

One of the key findings of this exercise was a recognition of the lack of knowledge/ 
expertise around effective practices in student learning assessment among the associate deans 
and the assessment coordinators. Many assessment coordinators, it turned out, were junior 
faculty members and their experience with student learning assessment, as well as that of the 
associate deans, was dependent upon their disciplinary backgrounds. For example, faculty 
and administrators within specialized accredited programs had more assessment knowledge 
and/or experience than faculty and administrators from programs without specialized 
accreditation. This lack of evenness in key constituents’ knowledge around assessment has 
also been identified as potentially problematic in other studies exploring program-level student 
learning assessment (Kelley, Tong, & Choi, 2010). In response to this concern, those leading 
the ASL initiative identified a rubric in the existing assessment literature and modified it 
(with permission) so that expectations were clear around the institutional expectation for 
program-level student learning assessment (Fulcher & Orem, 2010). In addition to the rubric, 
faculty and administrators were sent screencast videos made by the faculty associate that 
discussed key terms and provided explanations to help increase knowledge of student learning 
assessment. All academic programs were then asked to evaluate their plans using the adapted 
rubric. Initially, all programs were given the same deadline to evaluate all elements of their 
assessment plans (from the articulation of their outcomes to follow-up action plans). But, it 
quickly became clear that this approach would not be the best way to engage faculty in a 
thoughtful and reflective process. Some colleges needed more time than others, in part because 
they had more programs within their departments. Thus, different completion timelines were 
established but all participated in the same evaluation. 

Ensure a Level of Quality Control 

As noted above, the rubric adopted for this initiative was a modified version of an 
existing rubric (Fulcher & Orem, 2010). It was modified specifically to fit our institutional 
needs and captured what our assessment initiative leaders viewed as the core elements of an 
assessment plan regardless of discipline. These elements include: 
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Outcomes that are clearly measureable and indicate specifically the knowledge, skills, or 
attitudes that students are expected to have at a certain point. 

• Curriculum maps that specifically identify where program-level outcomes 
are introduced, practiced, and assessed. 

• Assessment measures that correspond to each outcome and include at least 
one direct measure of each outcome. 

• An appropriate rationale for each measure and measurement practice (i.e., 
information about the specific content of the measure or specific items of 

a larger measure that are used to indicate performance, about the reliability 
and validity of the measure, where appropriate, and about the context of the 
measure including course information, student level, etc.). 

• Criteria for success (e.g., benchmarking) for each measure of an outcome 
that includes a rationale for the selection of the specific criteria (e.g., 
program, discipline, or regional accrediting requirements). 

• Results that include analysis and interpretation of the data, as well as some 
specific follow-up action plans relevant to student learning (i.e., results that 
go well beyond the report of merely “criterion met” for all outcomes). 

Utilization of the rubric enabled the institution to ask faculty to step back from viewing 
assessment of student learning from primarily a quantitative perspective (i.e., in terms of how 
many students meet the criteria, or whether it was complete or not complete) and to focus 
instead on the quality of the process. Each component of the rubric had specific language aligned 
with numerical scores (1 to 4, with 1 identifying the element as absent and 4 representing the 
highest quality element) to enable programs to understand how to define quality for each 
core element. This is consistent with the approach endorsed by Fulcher and Orem (2010), 
who contend that the quality of the assessment process must be evaluated and considered 
if institutions are to guide programs toward improving student learning outcomes, which is, 
of course, the true purpose of student learning assessment. Additionally, the campus-wide 
rubric began to standardize the language of assessment across the institution and facilitated 
an increase in the assessment knowledge of individuals who were responsible for this work. 
Having clear descriptions of the core elements and their level of quality helped to bring faculty 
across campus to a common understanding of the assessment process, assessment language, 
and the expectations of the institution around student learning outcomes. 

Introduction and completion of the rubric by faculty was the first step to achieve 
an initial quality check; however, an institution-wide check was also necessary. This was 
important for two reasons. First, it was important to determine if program faculty were using 
the rubric correctly and if they were being honest in their evaluations. Second, it was critical to 
pinpoint the programmatic variability noted earlier within the institution as it pertained to the 
core elements. Having a more specific idea regarding variability would enable the institution to 
celebrate the disciplines and/or programs that were doing well and to appropriately address the 
programs that were not meeting expectations. To achieve these goals, the institution utilized 
several levels of review. 

First, the University Assessment Committee, including faculty representation from 
each of the five colleges as well as student affairs and student support services, were included 
in the review process. Their specific responsibility was to evaluate individual plans using the 
modified rubric. Random assignment of plans to committee members, along with use of an 
electronic assessment management system, allowed the process to be streamlined. In order 
for the committee members to evaluate the plans appropriately, norming sessions were held 
to establish inter-rater agreement. The second level of review was completed by the associate 
deans who scored all program plans within their respective colleges. All evaluators (committee 
members and associate deans) were instructed to provide specific qualitative feedback for any 
scores below 3.5 (on the 4-point scale) in order to provide direction for program improvement. 
The faculty associate was responsible for reviewing all plans and then compiled the results 
from the reviewers and the qualitative feedback. The reports were then sent to each of the 
respective programs. If programs received scores below 3.5, they were provided a time frame 
to make necessary modifications to their plan based on the feedback. The faculty associate 
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then went back in to review the plans a final time after programs had time to modify and the 
programs were scored again, producing a pre-ASL initiative and post-ASL initiative score. 

Using these scores, heat maps were developed. A heat map is a graphic representation 
of data where the values contained in a matrix are shown using colors. In the case of these 
heat maps, a high score of 4 was illustrated using the color green (good) and a low score 
of 1 was illustrated using the color red (needs improvement). All numbers in between are 
variations of those colors. Heat maps were developed using average element scores across 
the entire institution, within individual colleges, and for individual programs. Presentation of 
the heat maps at all three levels allowed faculty and administrators to see more clearly where 
improvement was needed and allowed them to create intervention procedures at multiple 
levels. Two example heat maps, with pre- and post-scores (i.e., before and after feedback 
was provided and plans were revised as part of the initiative), across all colleges/schools can 
be seen in Figure 1. Statistically significant differences (with improved scores) were found 
for at least some of the core elements across all colleges. The heat maps proved to be very 
beneficial in communicating a clear visual of the results of this work that was easy for all 
constituents to understand. In one college, an associate dean indicated that the heat maps 
were particularly effective in demonstrating areas that needed attention. Faculty did not 
enjoy viewing an area of red or orange amongst the green. Two assessment coordinators 
immediately asked, “What do we need to do to get out of the red and into the green?” 


Year 1ASL Initiative Scores, Fall 2013 


College 

Outcomes 

Curriculum 

Map 

Type of 
Measure 

Rationale 

for 

Measure 

Criteria for 
Success 

Results 

Action 

Plans 

A 

3.5 

3.7 

3.7 

3.5 

3.3 

3.0 

2.4 

B 

3.5 

2.8 

3.2 

2.7 

2.3 

2.0 

2.0 

C 

3.6 

4.0 

4.0 

4.0 

4.0 

3.8 

3.5 

D 

3.0 

2.8 

3.2 

2.6 

3.1 

1.7 

1.3 

E 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

F 

2.8 

3.3 

3.3 

2.1 

2.2 

2.6 

2.3 

OVERALL 

3.4 

3.4 

3.6 

3.1 

3.1 

3.4 

3.2 


Year 2 ASL Initiative Scores, Fall 2014 


College 

Outcomes 

Curriculum 

Map 

Type of 
Measure 

Rationale 

for 

Measure 

Criteria for 
Success 

Results 

Action 

Plans 

A 

3.6 

3.8 

3.7 

3.6 

3.5 



B 

3.8 

3.0 

3.2 

3.3* 

3.2* 



C 

4.0* 

4.0 

4.0 

4.0 

4.0 

4.0* 

4.0* 

D 

4.0* 

4.0* 

4.0 

4.0* 

4.0 

3.1* 

2.8* 

E 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

F 

3.8* 

3.7 

3.6* 

3.6* 

3.5* 

3.1 

2.9 

OVERALL 

3.8* 

3.7* 

3.8* 

3.7* 

3.6* 

3.7* 

3.6* 


Figure 1. College-Wide Heat Map. 

Note. *Year 2 means significantly different from Year 1 means (p < .05), gray cells 
post-measure not completed yet. 

Provide Appropriate Support and Resources 

This qualitative review process now occurs on an annual basis. We continue 
to see improvement in the quality of the core elements of assessment plans. One key 
reason this initiative continues to be successful is the continuing support of the academic 
administration. The provost’s budget includes line items for the purpose of supporting 
student learning assessment on campus. For example, assessment coordinators are provided 
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The process was 
beneficial for both 
university administrators 
and faculty. Academic 
administrators now 
have a clearer picture 
of the status of student 
learning assessment 
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some level of course release or summer funding in exchange for their service. The faculty 
associate is supported by a part-time alternate work assignment to coordinate this (and 
other) work, including chairing the university-wide assessment committee. Associate deans 
are charged with supervision of the assessment of student learning within their colleges and 
are evaluated, in part, on the degree to which programs within their colleges improve with 
regard to university expectations regarding the assessment of student learning. This level of 
support is crucial for the initiative to be successful. It is important to note, however, that this 
support was also in place prior to the implementation of the initiative. What really changed 
is that we implemented both a comprehensive plan for evaluating assessment quality and a 
comprehensive communication plan regarding the requirements of this plan. Those appear 
to be the key to the success of this initiative. 

Conclusion 

Assessment is about both the process and the end result (improving student learning 
outcomes). The ASL initiative described in this report was focused on process. It took a year 
and a half to produce the initial pre- and post-results described. The investment of time 
and resources was, we believe, a good one. The process was beneficial for both university 
administrators and faculty. Academic administrators now have a clearer picture of the status 
of student learning assessment across the campus. Specifically, they are now aware of the 
quality contained with the program assessment plan rather than knowing a plan exists with 
“x” number of outcomes, results, and action plans. And, faculty report that they have a better 
understanding of the expectations of the institution regarding student learning assessment 
and several of our specialized accredited programs feel this process has strengthened the 
efforts necessary to meet external mandates related to student learning assessment. There is 
clear alignment between the guiding principles suggested in the literature and the successful 
implementation of an ASL initiative at this regional comprehensive institution. These 
principles, as well as the experience of this IHE, provide a generalizable and practical model 
for other institutions who are interested in this approach to improving student learning. 
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